{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# python网络爬虫应用实战（第20期）第6课书面作业\n",
    "学号：115799\n",
    "\n",
    "**作业内容：**  \n",
    "1. 编程爬虫程序，尝试以统计学（https://baike.baidu.com/item/%E7%BB%9F%E8%AE%A1%E5%AD%A6/1175） 为起始页面，抓取100条百度百科词条页面，结果截图  \n",
    "2. 尝试爬取百度网络爬虫贴吧（https://tieba.baidu.com/f?kw=%E7%BD%91%E7%BB%9C%E7%88%AC%E8%99%AB&traceid=） 第一页中的所有帖子标题与帖子链接"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 第1题\n",
    "代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "webpages\\1.txt\n",
      "webpages\\2.txt\n",
      "webpages\\3.txt\n",
      "webpages\\4.txt\n",
      "webpages\\5.txt\n",
      "webpages\\6.txt\n",
      "webpages\\7.txt\n",
      "webpages\\8.txt\n",
      "webpages\\9.txt\n",
      "webpages\\10.txt\n",
      "webpages\\11.txt\n",
      "webpages\\12.txt\n",
      "webpages\\13.txt\n",
      "webpages\\14.txt\n",
      "webpages\\15.txt\n",
      "webpages\\16.txt\n",
      "webpages\\17.txt\n",
      "webpages\\18.txt\n",
      "webpages\\19.txt\n",
      "webpages\\20.txt\n",
      "webpages\\21.txt\n",
      "webpages\\22.txt\n",
      "webpages\\23.txt\n",
      "webpages\\24.txt\n",
      "webpages\\25.txt\n",
      "webpages\\26.txt\n",
      "webpages\\27.txt\n",
      "webpages\\28.txt\n",
      "webpages\\29.txt\n",
      "webpages\\30.txt\n",
      "webpages\\31.txt\n",
      "webpages\\32.txt\n",
      "webpages\\33.txt\n",
      "webpages\\34.txt\n",
      "webpages\\35.txt\n",
      "webpages\\36.txt\n",
      "webpages\\37.txt\n",
      "webpages\\38.txt\n",
      "webpages\\39.txt\n",
      "webpages\\40.txt\n",
      "webpages\\41.txt\n",
      "webpages\\42.txt\n",
      "webpages\\43.txt\n",
      "webpages\\44.txt\n",
      "webpages\\45.txt\n",
      "webpages\\46.txt\n",
      "webpages\\47.txt\n",
      "webpages\\48.txt\n",
      "webpages\\49.txt\n",
      "webpages\\50.txt\n",
      "webpages\\51.txt\n",
      "webpages\\52.txt\n",
      "webpages\\53.txt\n",
      "webpages\\54.txt\n",
      "webpages\\55.txt\n",
      "webpages\\56.txt\n",
      "webpages\\57.txt\n",
      "webpages\\58.txt\n",
      "webpages\\59.txt\n",
      "webpages\\60.txt\n",
      "webpages\\61.txt\n",
      "webpages\\62.txt\n",
      "webpages\\63.txt\n",
      "webpages\\64.txt\n",
      "webpages\\65.txt\n",
      "webpages\\66.txt\n",
      "webpages\\67.txt\n",
      "webpages\\68.txt\n",
      "webpages\\69.txt\n",
      "webpages\\70.txt\n",
      "webpages\\71.txt\n",
      "webpages\\72.txt\n",
      "webpages\\73.txt\n",
      "webpages\\74.txt\n",
      "webpages\\75.txt\n",
      "webpages\\76.txt\n",
      "webpages\\77.txt\n",
      "webpages\\78.txt\n",
      "webpages\\79.txt\n",
      "webpages\\80.txt\n",
      "webpages\\81.txt\n",
      "webpages\\82.txt\n",
      "webpages\\83.txt\n",
      "webpages\\84.txt\n",
      "webpages\\85.txt\n",
      "webpages\\86.txt\n",
      "webpages\\87.txt\n",
      "webpages\\88.txt\n",
      "webpages\\89.txt\n",
      "webpages\\90.txt\n",
      "webpages\\91.txt\n",
      "webpages\\92.txt\n",
      "webpages\\93.txt\n",
      "webpages\\94.txt\n",
      "webpages\\95.txt\n",
      "webpages\\96.txt\n",
      "webpages\\97.txt\n",
      "webpages\\98.txt\n",
      "webpages\\99.txt\n",
      "webpages\\100.txt\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "import datetime, random, re, os\n",
    "\n",
    "headers = {\n",
    "    'User-AGent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36'\n",
    "}\n",
    "\n",
    "visited_urls = set()\n",
    "unvisited_urls = set()\n",
    "save_dir = 'webpages'\n",
    "max_urls = 100 #抓取100条百度百科词条页面，可根据需要修改\n",
    "url_base = 'https://baike.baidu.com'\n",
    "url_start = '/item/%E7%BB%9F%E8%AE%A1%E5%AD%A6/1175'\n",
    "\n",
    "unvisited_urls.add(url_start)\n",
    "visited_count = 0\n",
    "\n",
    "def savepages(x):\n",
    "    url = url_base + x\n",
    "    html = requests.get(url=url, headers=headers)\n",
    "    pagename = os.path.join(save_dir, str(visited_count+1)+'.txt')\n",
    "    with open(pagename, 'w', encoding='utf-8') as f:\n",
    "        f.write(html.text)\n",
    "    print(pagename)\n",
    "    bsObj = BeautifulSoup(html.text, 'lxml')\n",
    "    for link in bsObj.findAll('a', href=re.compile(\"^(/item/)((?!:|/|\\?).)*$\")):\n",
    "        if 'href' in link.attrs:\n",
    "            unvisited_urls.add(link.attrs['href'])\n",
    "\n",
    "while visited_count < max_urls and len(unvisited_urls) > 0:\n",
    "    x = unvisited_urls.pop()\n",
    "    if x not in visited_urls:\n",
    "        savepages(x)\n",
    "        visited_count += 1\n",
    "        visited_urls.add(x)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "截图如下：\n",
    "[![j31Fot.png](https://s1.ax1x.com/2022/07/02/j31Fot.png)](https://imgtu.com/i/j31Fot)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 第2题\n",
    "1. 首先，百度网络爬虫贴吧（https://tieba.baidu.com/f?kw=%E7%BD%91%E7%BB%9C%E7%88%AC%E8%99%AB&traceid=）已经无法打开，截图如下：   \n",
    "[![j31DFx.png](https://s1.ax1x.com/2022/07/02/j31DFx.png)](https://imgtu.com/i/j31DFx)  \n",
    "2. 因此，作业里尝试打开昆虫贴吧（https://tieba.baidu.com/f?ie=utf-8&kw=%E6%98%86%E8%99%AB&fr=search）  \n",
    "\n",
    "代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'求助各位大佬，帮忙看看这个是什么虫子': 'https://tieba.baidu.com//p/7904756814'}\n",
      "{'这是什么昆虫': 'https://tieba.baidu.com//p/7904847960'}\n",
      "{'请问这是什么，北京': 'https://tieba.baidu.com//p/7904808506'}\n",
      "{'请大神告知图四、图五毛毛虫是何物': 'https://tieba.baidu.com//p/7904725756'}\n",
      "{'大家好，请问这是光端缺翅虎甲吗？谢谢！': 'https://tieba.baidu.com//p/7904518285'}\n",
      "{'请问大佬这是什么虫': 'https://tieba.baidu.com//p/7904799551'}\n",
      "{'这是什么虫？家里突然多起来了。谢谢': 'https://tieba.baidu.com//p/7904452752'}\n",
      "{'这是什么步甲的幼虫，怎么养': 'https://tieba.baidu.com//p/7904824430'}\n",
      "{'求问大佬这是果蝇吗': 'https://tieba.baidu.com//p/7904828937'}\n",
      "{'求鉴定，这是什么甲，坐标贵州省安顺市': 'https://tieba.baidu.com//p/7904820886'}\n",
      "{'问一下北京城里哪有卖昆虫标本的': 'https://tieba.baidu.com//p/7904750448'}\n",
      "{'问虫  请问这是什么金龟子什么蜘蛛': 'https://tieba.baidu.com//p/7903981539'}\n",
      "{'大佬们，这是什么萤': 'https://tieba.baidu.com//p/7904787185'}\n",
      "{'这些是什么？': 'https://tieba.baidu.com//p/7904778976'}\n",
      "{'这是什么蝴蝶，长着一个蜜蜂屁股': 'https://tieba.baidu.com//p/7904352129'}\n",
      "{'请教下吧里大佬这是什么蛾子？': 'https://tieba.baidu.com//p/7904321652'}\n",
      "{'这是什么虫子？': 'https://tieba.baidu.com//p/7899491842'}\n",
      "{'求问这个学名叫什么啊': 'https://tieba.baidu.com//p/7904530067'}\n",
      "{'这是天牛吗？': 'https://tieba.baidu.com//p/7904798099'}\n",
      "{'【求助】重发！求问这床上的小甲虫是啥！！': 'https://tieba.baidu.com//p/5783586976'}\n",
      "{'请教个蜻蜓，木蜂，蛛': 'https://tieba.baidu.com//p/7904743754'}\n",
      "{'这是哪种的蛾子': 'https://tieba.baidu.com//p/7904747398'}\n",
      "{'这是什么虫？跟螳螂好像': 'https://tieba.baidu.com//p/7902941769'}\n",
      "{'新人刚入坑求问': 'https://tieba.baidu.com//p/7904741281'}\n",
      "{'油斑马': 'https://tieba.baidu.com//p/7904786518'}\n",
      "{'家里出现的，有大佬知道是什么虫子吗': 'https://tieba.baidu.com//p/7904701435'}\n",
      "{'超级黑 杂交选优': 'https://tieba.baidu.com//p/7904784317'}\n",
      "{'救！！被虫子咬了，哪位大佬知道这是怎么回事吗？': 'https://tieba.baidu.com//p/7904712390'}\n",
      "{'这是什么虫啊，坐标浙江': 'https://tieba.baidu.com//p/7904767549'}\n",
      "{'这个是什么天牛？': 'https://tieba.baidu.com//p/7904768414'}\n",
      "{'问问吧友这是什么蟋螽': 'https://tieba.baidu.com//p/7904451071'}\n",
      "{'问一下这是什么虫啊在学校宿舍发现的': 'https://tieba.baidu.com//p/6162756447'}\n",
      "{'问虫及求助，睡着睡着有感觉抓到的': 'https://tieba.baidu.com//p/7903866714'}\n",
      "{'万能的8u们': 'https://tieba.baidu.com//p/7856873376'}\n",
      "{'虫子掉色了怎么回事？': 'https://tieba.baidu.com//p/7900628515'}\n",
      "{'请教下大佬这是什么蝽？': 'https://tieba.baidu.com//p/7904749911'}\n",
      "{'求助这是什么昆虫': 'https://tieba.baidu.com//p/7902590427'}\n",
      "{'这是个啥甲虫吃啥啊我想养': 'https://tieba.baidu.com//p/7904570625'}\n",
      "{'大佬们这是什么虫子': 'https://tieba.baidu.com//p/7904236178'}\n",
      "{'求助，北京这是什么天牛': 'https://tieba.baidu.com//p/7904675291'}\n",
      "{'问一个问题，蚂蚱和蝈蝈可以杂交吗': 'https://tieba.baidu.com//p/7904580836'}\n",
      "{'难受死我了': 'https://tieba.baidu.com//p/7904736556'}\n",
      "{'【其他方面】昆虫吧专用水楼由于各位吧友对于等级的需要 特开设': 'https://tieba.baidu.com//p/6954769095'}\n",
      "{'这是什么蜜蜂': 'https://tieba.baidu.com//p/7903931219'}\n",
      "{'蝽象求鉴定': 'https://tieba.baidu.com//p/7904711500'}\n",
      "{'兄弟们这是小强吗请问。。': 'https://tieba.baidu.com//p/7903749340'}\n",
      "{'【征集】征集一些笨蝗的照片与标本': 'https://tieba.baidu.com//p/7872959062'}\n",
      "{'这蛆一样的虫子叫什么': 'https://tieba.baidu.com//p/7904692100'}\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "from lxml import etree\n",
    "import datetime, random, re, os\n",
    "\n",
    "headers = {\n",
    "    'User-AGent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36'\n",
    "}\n",
    "url = 'https://tieba.baidu.com/f?ie=utf-8&kw=%E6%98%86%E8%99%AB&fr=search'\n",
    "r=requests.get(url,headers=headers)\n",
    "r_text=r.text.replace('-->','').replace('<!--','')\t\t\t\t#解决被注释掉的问题\n",
    "tree=etree.HTML(r_text)\n",
    "data_list=tree.xpath('//li[@class=\" j_thread_list clearfix thread_item_box\"]/div/div[2]/div[1]/div[1]/a')\n",
    "data_= [] \n",
    "\n",
    "for data in data_list:\n",
    "    data_dic={}\n",
    "    title=data.xpath('./text()')[0]\n",
    "    link='https://tieba.baidu.com/'+ data.xpath('./@href')[0]\n",
    "    data_dic[title]=link\n",
    "    data_.append(data_dic)\n",
    "\n",
    "for i in data_:\n",
    "    print(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "截图如下：  \n",
    "[![j33wjg.png](https://s1.ax1x.com/2022/07/02/j33wjg.png)](https://imgtu.com/i/j33wjg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "vscode": {
   "interpreter": {
    "hash": "9f35b62a1d17a2b36b9c54ddf6a1c189fdf51f8ec8cea898ba9fed29bc45b6fd"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
